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Abstract 

Formal  verification  and  validation  play  a  crucial  role  in  making  cyber-physical  systems  (CPS)  safe. 
Formal  methods  make  strong  guarantees  about  the  system  behavior  if  accurate  models  of  the  sys¬ 
tem  can  be  obtained,  including  models  of  the  controller  and  of  the  physical  dynamics.  In  CPS, 
models  are  essential;  but  any  model  we  could  possibly  build  necessarily  deviates  from  the  real 
world.  If  the  real  system  fits  to  the  model,  its  behavior  is  guaranteed  to  satisfy  the  correctness 
properties  verified  w.r.t.  the  model.  Otherwise,  all  bets  are  off.  This  paper  introduces  ModelPlex,  a 
method  ensuring  that  verification  results  about  models  apply  to  CPS  implementations.  ModelPlex 
provides  correctness  guarantees  for  CPS  executions  at  runtime:  it  combines  offline  verification  of 
CPS  models  with  runtime  validation  of  system  executions  for  compliance  with  the  model.  Model¬ 
Plex  ensures  that  the  verification  results  obtained  for  the  model  apply  to  the  actual  system  runs 
by  monitoring  the  behavior  of  the  world  for  compliance  with  the  model,  assuming  the  system  dy¬ 
namics  deviation  is  bounded.  If,  at  some  point,  the  observed  behavior  no  longer  complies  with  the 
model  so  that  offline  verification  results  no  longer  apply,  ModelPlex  initiates  provably  safe  fallback 
actions.  This  paper,  furthermore,  develops  a  systematic  technique  to  synthesize  provably  correct 
monitors  automatically  from  CPS  proofs  in  differential  dynamic  logic. 


1  Introduction 


Cyber-physical  systems  (CPS)  span  controllers  and  the  relevant  dynamics  of  the  environment. 
Since  safety  is  crucial  for  CPS,  their  models  (e.  g.,  hybrid  system  models  [29])  need  to  be  verified 
formally.  Formal  verification  guarantees  that  a  model  is  safe  w.r.t.  a  safety  property.  The  remaining 
task  is  to  validate  whether  those  models  are  adequate,  so  that  the  verification  results  transfer  to 
the  system  implementation  [16,  38].  This  paper  introduces  ModelPlex,  a  method  to  synthesize 
monitors  by  theorem  proving :  it  uses  sound  proof  rules  to  formally  verify  that  a  model  is  safe  and 
to  synthesize  provably  correct  monitors  that  validate  compliance  of  system  executions  with  that 
model. 

System  execution,  however,  provides  many  opportunities  for  surprising  deviations  from  the 
model:  faults  may  cause  the  system  to  function  improperly  [39],  sensors  may  deliver  uncertain 
values,  actuators  suffer  from  disturbance,  or  the  formal  verification  may  have  assumed  simpler 
ideal- world  dynamics  for  tractability  reasons  or  made  unrealistically  strong  assumptions  about  the 
behavior  of  other  agents  in  the  environment.  Simpler  models  are  often  better  for  real-time  deci¬ 
sions  and  optimizations,  because  they  make  predictions  feasible  to  compute  at  the  required  rate. 
The  same  phenomenon  of  simplicity  for  predictability  is  often  exploited  for  the  models  in  formal 
verification  and  validation.  As  a  consequence,  the  verification  results  obtained  about  models  of  a 
CPS  only  apply  to  the  actual  CPS  at  runtime  to  the  extent  that  the  system  fits  to  the  model. 

Validation ,  i.  e.,  checking  whether  a  CPS  implementation  fits  to  a  model,  is  an  interesting  but 
difficult  problem.  Even  more  so,  since  CPS  models  are  more  difficult  to  analyze  than  ordinary 
(discrete)  programs  because  of  the  physical  plant,  the  environment,  sensor  inaccuracies,  and  actu¬ 
ator  disturbance.  In  CPS,  models  are  essential;  but  any  model  we  could  possibly  build  necessarily 
deviates  from  the  real  world.  Still,  good  models  are  approximately  right,  i.  e.,  within  certain  error 
margins. 

In  this  paper,  we  settle  for  the  question  of  runtime  model  validation ,  i.  e.  validating  whether  the 
model  assumed  for  verification  purposes  is  adequate  for  a  particular  system  execution  to  ensure 
that  the  verification  results  apply  to  the  current  execution }  But  we  focus  on  verifiably  correct 
runtime  validation  to  ensure  that  verified  properties  of  models  provably  apply,  which  is  important 
for  safety  and  certification  [5]. 

If  the  observed  system  execution  fits  to  the  verified  model,  then  this  execution  is  safe  according 
to  the  offline  verification  result  about  the  model.  If  it  does  not  fit,  then  the  system  is  potentially 
unsafe  because  it  no  longer  has  an  applicable  safety  proof,  so  we  initiate  a  verified  fail-safe  action  to 
avoid  safety  risks.  Checking  whether  a  system  execution  fits  to  a  verified  model  includes  checking 
that  the  actions  chosen  by  the  (unverified)  controller  implementation  fit  to  one  of  the  choices  and 
requirements  of  the  verified  controller  model.  It  also  includes  checking  that  the  observed  states 
can  be  explained  by  the  plant  model.  The  crucial  questions  are:  How  can  a  compliance  monitor  be 
synthesized  that  provably  represents  the  verified  model?  How  much  safety  margin  does  a  system 
need  to  ensure  that  fail-safe  actions  are  initiated  early  enough  for  the  system  to  remain  safe  even  if 

1  ModelPlex  checks  system  execution  w.r.t.  a  monitor  specification,  and  thus,  belongs  to  the  field  of  runtime  verifica¬ 
tion  [16],  In  this  paper  we  use  the  term  runtime  validation  in  order  to  clearly  convey  the  purpose  of  monitoring  (i.  e., 
runtime  verification:  monitor  properties  without  offline  verification;  ModelPlex:  monitor  model  adequacy  to  transfer 
offline  verification  results). 
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its  behavior  ceases  to  comply  with  the  model? 

The  second  question  is  related  to  feedback  control  and  can  only  be  answered  when  assuming 
constraints  on  the  deviation  of  the  real  system  dynamics  from  the  plant  model  [33].  Otherwise,  i.  e., 
if  the  real  system  can  be  infinitely  far  off  from  the  model,  safety  guarantees  are  impossible.  By 
the  sampling  theorem  in  signal  processing  [37],  such  constraints  further  enable  compliance  mon¬ 
itoring  solely  on  the  basis  of  sample  points  instead  of  the  unobservable  intermediate  states  about 
which  no  sensor  data  exists.2  This  paper  presents  ModelPlex,  a  method  to  synthesize  verifiably 
correct  runtime  validation  monitors  automatically.  ModelPlex  uses  theorem  proving  with  sound 
proof  rules  [29]  to  turn  hybrid  system  models  into  monitors  in  a  verifiably  correct  way.  Upon 
noncompliance,  ModelPlex  initiates  provably  safe  fail-safe  actions.  System-level  challenges  w.r.t. 
monitor  implementation  and  violation  cause  diagnosis  are  discussed  elsewhere  [8,  19,  41]. 


2  Preliminaries:  Differential  Dynamic  Logic 

For  hybrid  systems  verification  we  use  differential  dynamic  logic  dC  [27,  29,  31],  which  has  a 
notation  for  hybrid  systems  as  hybrid  programs.  dC  allows  us  to  make  statements  that  we  want 
to  be  true  for  all  runs  of  a  hybrid  program  ([a]0)  or  for  at  least  one  run  ((a) o).  Both  constructs 
are  necessary  to  derive  safe  monitors:  we  need  [a](f>  proofs  so  that  we  can  be  sure  all  behavior  of 
a  model  (including  controllers)  are  safe;  we  need  (a)(p  proofs  to  find  monitor  specifications  that 
detect  whether  or  not  system  execution  fits  to  the  verified  model.  Table  1  summarizes  the  relevant 
syntax  fragment  of  hybrid  programs  together  with  an  informal  semantics.  The  semantics  p(a )  of 
hybrid  program  a  is  a  relation  on  initial  and  final  states  of  running  a  (defined  in  [27,  30]).  The 
set  of  dC  formulas  is  generated  by  the  following  grammar  (~  e  {<,<,=,>,>}  and  (f .  02  are 
arithmetic  expressions  in  /  over  the  reals): 

<t>  ::=  9i  ~  6*2  |  ->(j)  \  (j)  A  \  f  V  \  f  — >  ip  \  Vxf  |  3 xf  \  [a](j)  |  ( a)(j) 

2  When  such  constraints  are  not  available,  our  method  still  generates  verifiably  correct  runtime  tests,  which  detect 
deviation  from  the  model  at  the  sampling  points,  just  not  between  them.  A  fail-safe  action  will  then  lead  to  best-effort 
mitigation  of  safety  risks  (rather  than  guaranteed  safety). 

Table  1:  Hybrid  program  representations  of  hybrid  systems. 


Statement  Effect 

a]  (3  sequential  composition,  first  run  hybrid  program  a,  then  hybrid  program  8 

a  U  8  nondeterministic  choice,  following  either  hybrid  program  a  or  3 

a*  nondeterministic  repetition,  repeats  hybrid  program  a  n  >  0  times 

x  :=  9  assign  value  of  term  9  to  variable  x  (discrete  jump) 

x  :=*  assign  arbitrary  real  number  to  variable  x 

IF  check  that  a  particular  condition  F  holds,  and  abort  if  it  does  not 

(V,  =  (9 1 .... ,  evolve  x,  along  differential  equation  system  x\  =  0, 
x'n  =  9n  &  F)  restricted  to  maximum  evolution  domain  F 
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Figure  1:  Use  of  ModelPlex  monitors  along  a  system  execution 


Differential  dynamic  logic  comes  with  a  verification  technique  to  prove  correctness  properties 
of  hybrid  programs  (cf.  [31]  for  an  overview  of  d£  and  KeYmaera). 

3  ModelPlex  Approach  for  Verified  Runtime  Validation 

CPS  are  almost  impossible  to  get  right  without  sufficient  attention  to  prior  analysis,  for  instance  by 
formal  verification  and  formal  validation  techniques.  We  assume  to  be  given  a  verified  model  of  a 
CPS,  i.  e.  formula  (1)  is  proved  valid,3  for  example  using  [27,  31]. 


c p  — *  [a*]0  with  invariant  p  — *  [a]p  s.t.  0  — >  (p  and  p  — >  0 


(1) 


Formula  (1)  expresses  that  all  runs  of  the  hybrid  system  a*,  which  start  in  states  that  satisfy 
the  precondition  0  and  repeat  the  model  a  arbitrarily  many  times,  must  end  in  states  that  satisfy 
the  postcondition  0.  Formula  (1)  is  proved  using  some  form  of  induction,  which  shows  that  a  loop 
invariant  ip  holds  after  every  run  of  a  if  it  was  true  before.  The  model  a  is  a  hybrid  system  model 
of  a  CPS,  which  means  that  it  describes  both  the  discrete  control  actions  of  the  controllers  in  the 
system  and  the  continuous  physics  of  the  plant  and  the  system’s  environment. 

The  safety  guarantees  that  we  obtain  by  proving  formula  (1)  about  the  model  a*  transfer  to  the 
real  system,  if  the  actual  CPS  execution  fits  to  a*.  Since  we  want  to  preserve  safety  properties,  a 
CPS  7  fits  to  a  model  a*,  if  the  CPS  reaches  at  most  those  states  that  are  reachable  by  the  model, 
i.  e.,  p(j)  C  p(a*).  However,  we  do  not  know  7  and  therefore  need  to  find  a  condition  based  on  a* 
that  we  can  check  at  runtime  to  see  if  concrete  runs  of  7  behave  like  a*.  Checking  the  postcondition 
-0  is  not  sufficient  because,  if  0  does  not  hold,  the  system  is  already  unsafe.  Checking  the  invariant 
p  is  insufficient  as  well,  because  if  (p  does  not  hold  the  controller  can  no  longer  guarantee  safety, 
even  though  the  system  may  not  yet  be  unsafe.  But  if  we  detect  when  a  CPS  is  about  to  deviate 
from  a*  before  leaving  p,  we  can  still  switch  to  a  fail-safe  controller  to  avoid  -i0  from  happening. 

ModelPlex  derives  three  kinds  of  monitors  (model  monitor,  controller  monitor,  and  prediction 
monitor,  cf.  Fig.  1).  We  check  reachability  between  consecutive  states  in  a,  actri,  and  o()P|ant  by 
verifying  states  during  execution  against  the  corresponding  monitor. 

3  We  use  differential  dynamic  logic  (cLC)  and  KeYmaera  as  a  theorem  prover  to  illustrate  our  concepts  throughout  this 
paper.  The  concept  of  ModelPlex  is  not  predicated  on  the  use  of  KeYmaera  to  prove  (1).  Other  verification  techniques 
could  be  used  to  establish  validity  of  this  formula.  The  flexibility  of  the  underlying  logic  cLC,  its  support  for  both  [a]4> 
and  (a)0,  and  its  proof  calculus,  however,  are  exploited  for  systematically  constructing  monitors  from  proofs  in  the 


sequel. 
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Model  monitor  In  each  state  ut  we  test  the  sample  point  z/,_ |  from  the  previous  execution  7* _i 
for  deviation  from  the  single  a ,  not  a*  i.  e.,  test  (17-1,  vf)  G  p(a).  If  violated,  other  verified 
properties  may  no  longer  hold  for  the  system;  the  system,  however,  is  still  safe  if  a  prediction 
monitor  was  satisfied  on  v%-  \ .  Frequent  violations  indicate  an  inadequate  model  that  should 
be  revised  to  better  reflect  reality. 

Controller  monitor  In  intermediate  state  z/,  we  test  the  current  controller  decisions  of  the  im¬ 
plementation  7ctri  for  compliance  with  the  model,  i.  e.,  test  (z/,,  z/;)  G  p(actri).  Controller 
monitors  are  designed  for  switching  between  controllers  similar  to  Simplex  [36].  If  violated, 
the  commands  from  a  fail-safe  controller  replace  the  current  controller’s  decisions  to  ensure 
that  no  unsafe  commands  are  ever  actuated. 

Prediction  monitor  In  intermediate  state  z/j  we  test  the  worst-case  safety  impact  of  the  current 
controller  decisions  w.r.t.  the  predictions  of  a  bounded  deviation  plant  model  a«5piant,  which 
has  a  tolerance  around  the  model  plant  apiant,  i.  e.,  check  ui+ 1  |=  p  for  all  \  such  that 
(z>j,z/j+1)  G  /z(Q'rtpiant)-  Note,  that  we  simultaneously  check  all  isi+1  by  checking  9,  for  a 
characterizing  condition  of  Q^piam-  If  violated,  the  current  control  choice  is  not  guaranteed  to 
keep  the  system  safe  until  the  next  control  cycle  and,  thus,  a  fail-safe  controller  takes  over. 

The  assumption  for  the  prediction  monitor  is  that  the  real  execution  is  not  arbitrarily  far  off 
the  plant  models  used  for  safety  verification,  because  otherwise  guarantees  can  be  neither  made 
on  unobservable  intermediate  states  nor  on  safety  of  the  future  system  evolution  [33].  We  propose 
separation  of  disturbance  causes  in  the  models:  ideal  plant  models  Qp|anl  for  correctness  verifica¬ 
tion  purposes,  implementation  deviation  plant  models  a^iam  for  monitoring  purposes.  We  support 
any  deviation  model  (e.  g.,  piecewise  constant  disturbance,  differential  inclusion  models  of  distur¬ 
bance),  as  long  as  the  deviation  is  bounded  and  differential  invariants  can  be  found.  We  further 
assume  that  monitor  evaluations  are  at  most  some  e  time  units  apart  (e.  g.,  along  with  a  recurring 
controller  execution).  Note  that  disturbance  in  a<5piant  is  more  manageable  compared  to  a*,  because 
we  can  focus  on  single  runs  a  instead  of  repetitions  for  monitoring. 

3.1  Relation  between  States 

We  systematically  derive  a  check  that  inspects  states  of  the  actual  CPS  to  detect  deviation  from  the 
model  a*.  We  first  establish  a  notion  of  state  recall  and  show  that,  when  all  previous  state  pairs 
complied  with  the  model,  compliance  of  the  entire  execution  can  be  checked  by  checking  the  latest 
two  states  (z7_  1,  z/,)  (see  App.  A  for  proofs). 

Definition  1  (State  recall).  We  use  V  to  denote  the  set  of  variables  whose  state  we  want  to  recall. 
We  use  Yy  =  f\xeV  x  =  x  to  express  a  characterization  of  the  values  of  variables  in  a  state  prior 
to  a  run  of  a,  where  we  always  assume  the  fresh  variables  x~  to  occur  solely  in  Tf.  The  variables 
in  x  can  be  used  to  recall  this  state.  Likewise,  we  use  Yy  =  /\  x£V  x  =  x+  to  characterize  the 
posterior  states  and  expect  fresh  x+. 

With  this  notation  the  following  lemma  states  that  an  interconnected  sequence  of  a  transitions 
forms  a  transition  of  a*. 


4 


Lemma  1  (Loop  prior  and  posterior  state).  Let  a  be  a  hybrid  program  and  a*  be  the  program  that 
repeats  a  arbitrarily  many  times.  Assume  that  all  consecutive  pairs  of  states  (z/j_ i,  vf)  G  p(a)  of 
n  G  N+  executions,  whose  valuations  are  recalled  with  Tf  =  /\xeV  x  =  xl  and  T(7 1  are  plausible 
w.r.t.  the  model  a,  i.  e.,  \=  /\]<1<n  (Y^1  — >  (a)  Y).,]  with  Tf  =  T({-  and  Y y  =  Y y.  Then,  the 
sequence  of  states  originates  from  an  a*  execution  from  Y^  to  Y  y,  i.  e.,  |=  Tf  -G  (a*)Ty. 

Lemma  1  enables  us  to  check  compliance  with  the  model  a*  up  to  the  current  state  by  checking 
reachability  of  a  posterior  state  from  a  prior  state  on  each  execution  of  a  (i.  e.,  online  monitor¬ 
ing  [16],  which  is  easier  because  the  loop  was  eliminated).  To  find  compliance  checks  systemati¬ 
cally,  we  construct  formula  (2),  which  relates  a  prior  state  of  a  CPS  to  its  posterior  state  through  at 
least  one  path  through  the  model  a.  4 


t-v  ->  <a)T+  (2) 

This  formula  is  satisfied  in  a  state  u,  if  there  is  at  least  one  run  of  the  model  a  starting  in 
the  state  v  recalled  by  Tv  and  results  in  a  state  u  recalled  using  Y y.  In  other  words,  at  least 
one  path  through  a  explains  how  the  prior  state  v  got  transformed  into  the  posterior  state  u.  The 
d£  formula  (2)  characterizes  the  state  transition  relation  of  the  model  a  directly.  Its  violation 
witnesses  compliance  violation.  Compliance  at  all  intermediate  states  cannot  be  observed  by  real- 
world  sensors,  see  Section  3.5. 

In  principle,  formula  (2)  would  be  a  monitor,  because  it  relates  a  prior  state  to  a  posterior 
state  through  the  model  of  a  CPS;  but  the  formula  is  hard  if  not  impossible  to  evaluate  at  runtime, 
because  it  refers  to  a  hybrid  system  a,  which  includes  nondeterminism  and  differential  equations. 
The  basic  observation  is  that  any  formula  that  is  equivalent  to  (2)  but  conceptually  easier  to  evaluate 
in  a  state  would  be  a  correct  monitor.  We  use  theorem  proving  for  simplifying  formula  (2)  into 
quantifier- free  first-order  real  arithmetic  form  so  that  it  can  be  evaluated  efficiently  at  runtime.  The 
resulting  first-order  real  arithmetic  formula  can  be  easily  implemented  in  a  runtime  monitor  and 
executed  along  with  the  actual  controller.  A  monitor  is  executable  code  that  only  returns  true  if 
the  transition  from  the  prior  system  state  to  the  posterior  state  is  compliant  with  the  model.  Thus, 
deviations  from  the  model  can  be  detected  at  runtime,  so  that  appropriate  fallback  and  mitigation 
strategies  can  be  initiated. 

Remark  1.  The  complexity  for  evaluating  an  arithmetic  formula  over  the  reals  for  concrete  num¬ 
bers  is  linear  in  the  formula  size,  as  opposed  to  deciding  the  validity  of  such  formulas,  which  is 
doubly  exponential.  Evaluating  the  same  formula  on  floating  point  numbers  is  inexpensive,  but 
may  yield  wrong  results  due  to  rounding  errors;  on  exact  nationals  the  bit-complexity  can  be  non- 
negligible.  We  use  interval  arithmetic  to  obtain  reliable  results  efficiently  (cf.  App.  C). 

Example  1.  We  will  use  a  simple  water  tank  as  a  running  example  to  illustrate  the  concepts 
throughout  this  section.  The  water  tank  has  a  current  level  x  and  a  maximum  level  m.  The  water 
tank  controller,  which  runs  at  least  every  e  time  units,  nondeterministically  chooses  any  flow  f 
between  a  maximum  outflow  —1  and  a  maximum  inflow  This  water  tank  never  overflows,  as 
witnessed  by  a  proof  for  the  following  d£  formula. 

4  Consecutive  states  for  a*  mean  before  and  after  executions  of  a  (i.  e.,  or, or,  a,  not  within  a). 
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0<x<m/\e>0  — ► 


(/:=*;  ?(-l<f  <"=*);  t 

t  :=  0;  (x'  =  /,  t'  =  1  &  x  >  0  A  t  <  e))  *  (0  <  a:  <  m) 


3.2  ModelPlex  Monitor  Synthesis 

This  section  introduces  the  nature  of  ModelPlex  monitor  specifications,  our  approach  to  generate 
such  specifications  from  hybrid  system  models,  and  how  to  turn  those  specifications  into  monitor 
code  that  can  be  executed  at  runtime  along  with  the  controller. 

A  ModelPlex  specification  corresponds  to  the  d£  formula  (2).  If  the  current  state  of  a  sys¬ 
tem  does  not  satisfy  a  ModelPlex  specification,  some  behavior  that  is  not  reflected  in  the  model 
occurred  (e.  g.,  the  wrong  control  action  was  taken,  unanticipated  dynamics  in  the  environment  oc¬ 
curred,  sensor  uncertainty  led  to  unexpected  values,  or  the  system  was  applied  outside  the  specified 
operating  environment). 

A  model  monitor  Xm  checks  that  two  consecutive  states  v  and  uj  can  be  explained  by  an  execu¬ 
tion  of  the  model  a,  i.  e.,  (z/,  c u)  e  p(a).  In  the  sequel,  BV(a )  are  bound  variables  in  a,  FV( 0) 
are  free  variables  in  -0,  £  is  the  set  of  all  variables,  and  A\B  denotes  the  set  of  variables  being  in 
some  set  A  but  not  in  some  other  set  B.  Furthermore,  we  use  u\a  to  denote  v  projected  onto  the 
variables  in  A. 

Theorem  1  (Model  monitor  correctness).  Let  a*  be  provably  safe,  so  f=  0  — >  Let  Vrn  = 

BV ( a )  U  FV  (0).  Let  z/0,  ui,  u2,  u3  . . .  6  Mn  be  a  sequence  of  states,  with  u0  \—  f  and  that  agree 
on  H\Vm,  i.  e.,  z/o|£\vm  =  uk\ Y\Vmfor  all  k.  We  define  (u,  ul+ 1 )  |=  \m  as  Xm  evaluated  in  the  state 
resulting  from  v  by  interpreting  x+  as  vi+1{x)  for  all  x  G  Vm,  i.  e.,  vvf^x)  |=  Xm-  If  fa,  F+i)  \=  Xm 
for  all  i  <  n  then  we  have  vn\=  where 

Xm  =  (0| const  («)TVm)  (3) 

and  0| const  denotes  the  conditions  of  0  that  involve  only  constants  that  do  not  change  in  a,  i.  e., 
FVfa\ 

const  )  n  BV(a)  =  0. 

Our  approach  to  generate  monitor  specifications  from  hybrid  system  models  takes  a  verified  d£ 
formula  (1)  as  input  and  produces  a  monitor  in  quantifier- free  first-order  form  as  output.  The 
algorithm,  listed  in  App.  D,  involves  the  following  steps: 

1.  A  d£  formula  (1)  about  a  model  a  of  the  form  0  — »  [a*]fi>  is  turned  into  a  specification 
conjecture  (3)  of  the  form  0|COnst  — >  (a)Ty  . 

2.  Theorem  proving  on  the  specification  conjecture  (3)  is  applied  until  no  further  proof  rules 
are  applicable  and  only  first-order  real  arithmetic  formulas  remain  open. 

3.  The  monitor  specification  Xm  is  the  conjunction  of  the  unprovable  first-order  real  arithmetic 
formulas  from  open  sub-goals. 
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Generate  the  monitor  conjecture.  We  map  dC  formula  (1)  syntactically  to  a  specification  con¬ 
jecture  of  the  form  (3).  By  design,  this  conjecture  will  not  be  provable.  But  the  unprovable 
branches  of  a  proof  attempt  will  reveal  information  that,  had  it  been  in  the  premises,  would  make 
(3)  provable.  Through  Y0  ,  those  unprovable  conditions  collect  the  relations  of  the  posterior  state 
of  model  a  characterized  by  x+  to  the  prior  state  x,  i.  e.,  the  conditions  are  a  representation  of  (2) 
in  quantifier-free  first-order  real  arithmetic. 

Example  2.  The  specification  conjecture  for  the  water  tank  model  is  given  below.  It  is  constructed 
from  the  model  by  removing  the  loop,  flipping  the  modality,  and  formulating  the  specification 
requirement  as  a  property,  since  we  are  interested  in  a  relation  between  two  consecutive  states  v 
and  oj  ( recalled  by  x+,  f+  and  t+).  Using  theorem  proving  [34],  we  analyze  the  conjecture  to 
reveal  the  actual  monitor  specification. 

01™™,  t:=0;  (x'  =  /,  t'  =  1  k.  x  >  t)  At  <  e)^j  [x  =  x+  A  f  =  f  +  At  =  t+) 


Use  theorem  proving  to  analyze  the  specification  conjecture.  We  use  the  proof  rules  of  d£  [27, 
31]  to  analyze  the  specification  conjecture  ym.  These  proof  rules  syntactically  decompose  a  hybrid 
model  into  easier-to-handle  parts,  which  leads  to  sequents  with  first-order  real  arithmetic  formulas 
towards  the  leaves  of  a  proof.  Using  real  arithmetic  quantifier  elimination  we  close  sequents  with 
logical  tautologies,  which  do  not  need  to  be  checked  at  runtime  since  they  always  evaluate  to  true 
for  any  input.  The  conjunction  of  the  remaining  open  sequents  is  the  monitor  specification;  it 
implies  (2). 

A  complete  sequence  of  proof  rules  applied  to  the  monitor  conjecture  of  the  water  tank  is 
described  in  App.B.  Most  steps  are  simple  when  analyzing  specification  conjectures:  sequential 
composition  ((; )),  nondeterministic  choice  (<U)),  deterministic  assignment  ((:=))  and  logical  con¬ 
nectives  (Ar  etc.)  replace  current  facts  with  simpler  ones  or  branch  the  proof  (cf.  rules  in  [27,  30]). 
Challenges  arise  from  handling  nondeterministic  assignment  and  differential  equations  in  hybrid 
programs. 

Let  us  first  consider  nondeterministic  assignment  x  :=  *.  The  proof  rule  for  nondeterministic 
assignment  ((*))  results  in  a  new  existentially  quantified  variable.  By  sequent  proof  rule  3r,  this 
existentially  quantified  variable  is  instantiated  with  an  arbitrary  term  6,  which  is  often  a  new  logical 
variable  that  is  implicitly  existentially  quantified  [27].  Weakening  (Wr)  removes  facts  that  are  no 
longer  necessary. 


«»» 


3X(x:=X)<p1 
(x  :=  *)0 


T  h  0(0),  3xfi(x),  A  2 
T  h  3xfl(x),  A 


(Wr) 


r  h  a 
r  h  0,  a 


1  A  is  a  new  logical  variable 

1  9  is  an  arbitrary  term,  often  a  new  (existential)  logical  variable  X. 


Optimization  1  (Instantiation  Trigger).  If  the  variable  is  not  changed  in  the  remaining  a,  xr  =  xf 
is  in  T y  and  X  is  not  bound  in  Y(-  ,  then  instantiate  the  existential  quantifier  by  rule  3 r  with  the 
corresponding  xf  that  is  part  of  the  specification  conjecture  ( i.  e.,  9  =  xf),  since  subsequent  proof 
steps  are  going  to  reveal  6  =  xf  anyway. 
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Otherwise,  we  introduce  a  new  logical  variable,  which  may  result  in  an  existential  quantifier  in 
the  monitor  specification  if  no  further  constraints  can  be  found  later  in  the  proof. 

Example  3.  The  corresponding  steps  in  the  water  tank  proof  use  (*}  for  the  nondeterministic  flow 
assignment  (f  :—*)  and  3 r  to  instantiate  the  resulting  existential  quantifier  3 F  with  a  new  logical 
variable  F  ( plant  is  an  abbreviation  for  x'  —  /'.  t'  =  1  &  0  <  x  At  <  e).  We  show  the  proof 
without  and  with  application  of  Opt.  1. 

0  b  (f:=F)(?~ 1  <  /  <  r^){plant) T+  0PL  1  0  b  (/:=/+) 

3r’Wr0b  3F(f:=F)(?-l  <  f  <  ^){plant)r+l  <?-l  <f<  (plant) T+ 

'*:o-  (/:-*:?  I  <  ./  <  ;  plant  T  ' 

with  Opt.  1  (anticipate  /  =  f+  from  Y+) 


Next,  we  handle  differential  equations.  Even  when  we  can  solve  the  differential  equation, 
existentially  and  universally  quantified  variables  remain.  Let  us  inspect  the  corresponding  proof 
rule  from  the  d£  calculus  [31]. 


(0) 


3T>0  ((V0<f<T  {x  :=  y{i))H)  A  {x  :=  y(T))<j>)  , 
(x1 2  =  9  8zH)(j) 


(QE) 


QE(0)  2 

0 


1  T  and  t  are  fresh  logical  variables  and  (x  :=y(T ))  is  the  discrete  assignment  belonging  to  the  solution  y  of  the 
differential  equation  with  constant  symbol  x  as  symbolic  initial  value 

2  iff  0  =  QE(0),  0  is  a  first-order  real  arithmetic  formula,  QE(0)  is  an  equivalent  quantifier- free  formula  computable 
by  [7] 


For  differential  equations  we  have  to  prove  that  there  exists  a  duration  t,  such  that  the  differ¬ 
ential  equation  stays  within  the  evolution  domain  H  throughout  all  intermediate  times  t  and  the 
result  satisfies  0  at  the  end.  At  this  point  we  have  three  options: 

•  we  can  instantiate  the  existential  quantifier,  if  we  know  that  the  duration  will  be  t+; 

•  we  can  introduce  a  new  logical  variable,  which  is  the  generic  case  that  always  yields  correct 
results,  but  may  discover  monitor  specifications  that  are  harder  to  evaluate; 

•  we  can  use  quantifier  elimination  (QE)  to  obtain  an  equivalent  quantifier-free  result  (a  pos¬ 
sible  optimization  could  inspect  the  size  of  the  resulting  formula). 

Example  4.  In  the  analysis  of  the  water  tank  example,  we  solve  the  differential  equation  (see  (')) 
and  apply  the  substitutions  f  :=  F  and  t  :=  0.  In  the  next  step  (see  3 r,Wr),  we  instantiate  the 
existential  quantifier  3 T  with  t+  (i.  e.,  we  choose  T  =  t+  using  Opt.  1  with  the  last  conjunct)  and 
use  weakening  right  (Wr)  to  systematically  get  rid  of  the  existential  quantifier  that  would  otherwise 
still  be  left  around  by  rule  3 r.  Finally,  we  use  quantifier  elimination  (QE)  to  reveal  an  equivalent 
quantifier- free  formula. 

The  analysis  of  the  specification  conjecture  finishes  with  collecting  the  open  sequents  from  the 

dcf 

proof  to  create  the  monitor  specification  ym  =  f\(ppen  sequent).  The  collected  open  sequents  may 
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cj)\-  F  =  f  +  A  x+  =  x  +  Ft+  At+>OAx>OAs>t+>OA  Ft+  +  x  >  0 
QE  0  h  VO<f<T  (x  +  f+t  >  0  At  <  e)  A  F  =  f+  A  x+  =  x  +  Ft+  A  t+  =  t+ 

3r’Wr</>  h  3T>0((V0<£<T  (x  +  f+t  >  0  A  t  <  e))  A  F  =  f+  A  (x+  =  x  +  FT  A  t+  =  T)) 
^  </>  h  (/  :=  F;  t  :=  0} {{x'  =  f,  t'  =  1  &  x  >  0  A  t  <  e})T+ 


include  new  logical  variables  and  new  (Skolem)  function  symbols  that  were  introduced  for  non- 
deterministic  assignments  and  differential  equations  when  handling  existential  or  universal  quan¬ 
tifiers.  We  use  the  invertible  quantifier  rule  i3  to  re-introduce  existential  quantifiers  for  the  new 
logical  variables  (universal  quantifiers  for  function  symbols,  see  [27]  for  calculus  details).  Often, 
the  now  quantified  logical  variables  are  discovered  to  be  equal  to  one  of  the  post- state  variables 
later  in  the  proof,  because  those  variables  did  not  change  in  the  model  after  the  assignment.  If  this 
is  the  case,  we  can  use  proof  rule  3er  to  further  simplify  the  monitor  specification  by  substituting 
the  corresponding  logical  variable  x  with  its  equal  term  6. 

„  £  1-  3A-  (A,  ($<  I-*.)),  A  m 

T,  b  A  ■■■  r,$n  b  3x  (x  =  9  A  4>(x)) 

1  Among  all  open  branches,  free  logical  variable  X  only  occurs  in  the  branches  T,  <t>i  b  \Eq,  A 

2  Logical  variable  x  does  not  appear  in  term  6 


Example  5.  The  two  open  sequents  of  Examples  3  and  4  use  a  new  logical  variable  F  for  the 
nondeterministic  flow  assignment  f  ■=  *.  After  further  steps  in  the  proof  the  assumptions  reveal 
additional  information  F  =  f+.  Thus,  we  re-introduce  the  existential  quantifier  over  all  the  open 
branches  (73)  and  substitute  f+  for  F  (3cr).  The  sole  open  sequent  of  this  proof  attempt  is  the 
monitor  specification  \m  of  the  water  tank  model. 

(j)  b  -1  <  /+  <  A  x+  =  x  +  f+t+  Af+>0Ax>0Ae>f+>0A  f+t+  +  x  >  0 
b  3F(-1  <  F  <  ^  A  F  =  f  +  A  x+  =  x  +  Ft+  At+  >  0  A  x  >  0  A  e  >  t+  >  0  A  Ft+  +  x  >  0) 
a<t>  h  -1  <  F  <  ^  £  </>bF  =  /+Ax+=x  +  Ft+  Af+>0Ax>0Ae>f+>0A  Ft+  +  x  >  0 

3.3  Controller  Monitor  Synthesis 

A  controller  monitor  yc  checks  that  two  consecutive  states  v  and  u  are  reachable  with  one  con¬ 
troller  execution  ctctrl,  i.  e.,  (z/,  u)  e  p(ac tri)  with  Vc  =  BV(actrl)  U  FV(f>).  We  systematically 
derive  controller  monitors  from  formulas  0 1 const  — >  (ct ctri)^^,.  A  controller  monitor  can  be  used  to 
initiate  controller  switching  similar  to  Simplex  [36]. 

Theorem  2  (Controller  monitor  correctness).  Let  a  of  the  canonical  form  actri;  apiant.  Assume 
| =  0  — )•  \a*]fi)  has  been  proven  with  invariant  p  as  in  (1).  Let  u  |=  (f)\comt  A  p,  as  checked  by  Xm 
(Theorem  1).  Furthermore,  let  D  be  a  post-controller  state.  If  (u,  b)  |=  \c  with  Xc  =  0 \ const  — >■ 
(actri)Tyc  then  we  have  that  [y,  v)  G  p(otctrj )  and  u  |=  p. 
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3.4  Monitoring  in  the  Presence  of  Expected  Uncertainty  and  Disturbance 

Up  to  now  we  considered  exact  ideal-world  models.  But  real-world  clocks  drift,  sensors  measure 
with  some  uncertainty,  and  actuators  are  subject  to  disturbance.  This  makes  the  exact  models  safe 
but  too  conservative,  which  means  that  monitors  for  exact  models  are  likely  to  fall  back  to  a  fail¬ 
safe  controller  rather  often.  In  this  section  we  discuss  how  we  find  ModelPlex  specifications  so 
that  the  safety  property  (1)  and  the  monitor  specification  become  more  robust  to  expected  uncer¬ 
tainty  and  disturbance.  That  way,  only  unexpected  deviations  beyond  those  captured  in  the  normal 
operational  uncertainty  and  disturbance  of  a*  cause  the  monitor  to  initiate  fail-safe  actions. 

In  d C,  we  can,  for  example,  use  nondeterministic  assignment  from  an  interval  to  model  sensor 
uncertainty  and  piecewise  constant  actuator  disturbance  (e.  g.,  as  in  [24]),  or  differential  inequali¬ 
ties  for  actuator  disturbance  (e.  g.,  as  in  [35]).  Such  models  include  nondeterminism  about  sensed 
values  in  the  controller  model  and  often  need  more  complex  physics  models  than  differential  equa¬ 
tions  with  polynomial  solutions. 

Example  6.  We  incorporate  clock  drift,  sensor  uncertainty  and  actuator  disturbance  into  the  wa¬ 
ter  tank  model  to  express  expected  deviation.  The  measured  level  xs  is  within  a  known  sensor 
uncertainty  u  of  the  reed  level  x  (i.e.  xs  G  [x  —  u,x  +  u\).  We  use  differential  inequalities  to  model 
clock  drift  and  actuator  disturbance.  The  clock,  which  wakes  the  controller,  is  slower  than  the  real 
time  by  at  most  a  time  drift  of  c;  it  can  be  arbitrarily  fast.  The  water  flow  disturbance  is  at  most 
d,  but  the  water  tank  is  allowed  to  drain  arbitrarily  fast  (even  leaks  when  the  pump  is  on).  To 
illustrate  different  modeling  possibilities,  we  use  additive  clock  drift  and  multiplicative  actuator 
disturbance. 


0<x<toA£>0A  c<lA0<«A0<d 

(  xs  :=  *;  ?  (x  -  u  <  xs  <  x  +  u) ;  /:=*;?  (-1  <  /  < 


m—xs—u 

de 


(1  -c)); 


t  :=  0;  {x'  <  fd ,  l-c<t'&s>0At<  er})  *  (0  <  x  <  m) 


We  analyze  Example  6  in  the  same  way  as  the  previous  examples,  with  the  crucial  exception 
of  the  differential  inequalities.  We  cannot  use  the  proof  rule  (')  to  analyze  this  model,  because 
differential  inequalities  do  not  have  polynomial  solutions.  Instead,  we  use  the  DR  and  DE  proof 
rules  of  duC  [28,  29]  to  turn  differential  inequalities  into  a  differential- algebraic  constraint  form  that 
lets  us  proceed  with  the  proof.  Rule  DE  turns  a  differential  inequality  x'  <  6  into  a  quantified 
differential  equation  3 d(x'  =  d  k.  d  <  6)  with  an  equivalent  differential-algebraic  constraint. 
Rule  DR  turns  a  differential- algebraic  constraint  §  into  another  differential-algebraic  constraint 
<2),  which  implies  S\  written  — *  <§,  as  defined  in  [28]  (cf.  App.  B.l  for  an  example). 


(DR) 


-a  S  j 


(DE) 


VX(3  d(X  =  dAd<6AH)^X<9AH) 

(3 d(x'  =  d&i  d  <  0  A  H))<f  2 
(x'  <  0&cH)<f> 


1  differential  refinement:  differential-algebraic  constraints  A  have  the  same  changed  variables 

2  differential  inequality  elimination:  special  case  of  DR,  which  rephrases  the  differential  inequalities  <  as  differential- 
algebraic  constraints  (accordingly  for  other  or  mixed  inequalities  systems). 


Currently,  for  finding  model  monitors  our  prototype  tool  solves  differential  equations  by  the 
proof  rule  (').  Thus,  it  finds  model  monitor  specifications  for  differential  algebraic  equations  with 
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polynomial  solutions  and  for  differential  algebraic  inequalities,  which  can  be  refined  into  solvable 
differential  algebraic  equations  as  in  Example  6.  For  prediction  monitors  (discussed  in  Section  3.5) 
we  use  d£  techniques  for  finding  differential  variants  and  invariants,  differential  cuts  [28],  and 
differential  auxiliaries  [32]  to  handle  differential  equations  and  inequalities  without  polynomial 
solutions. 

3.5  Monitoring  Compliance  Guarantees  for  Unobservable  Intermediate  States 

With  controller  monitors,  non-compliance  of  a  controller  implementation  w.r.t.  the  modeled  con¬ 
troller  can  be  detected  right  away.  With  model  monitors,  non-compliance  of  the  actual  system 
dynamics  w.r.t.  the  modeled  dynamics  can  be  detected  when  they  first  occur.  We  switch  to  a  fail¬ 
safe  action,  which  is  verified  using  standard  techniques,  in  both  non-compliance  cases.  The  crucial 
question  is:  can  such  a  method  always  guarantee  safety?  The  answer  is  linked  to  the  image  com¬ 
putation  problem  in  model  checking  (i.  e.,  approximation  of  states  reachable  from  a  current  state), 
which  is  known  to  be  not  semi-decidable  by  numerical  evaluation  at  points;  approximation  with 
uniform  error  is  only  possible  if  a  bound  is  known  for  the  continuous  derivatives  [33].  This  im¬ 
plies  that  we  need  additional  assumptions  about  the  deviation  between  the  actual  and  the  modeled 
continuous  dynamics  to  guarantee  compliance  for  unobservable  intermediate  states.  Unbounded 
deviation  from  the  model  between  sample  points  just  is  unsafe,  no  matter  how  hard  a  controller 
tries.  Hence,  worst-case  bounds  capture  how  well  reality  is  reflected  in  the  model. 

We  derive  a  prediction  monitor  to  check  whether  a  current  control  decision  will  be  able  to 
keep  the  system  safe  for  time  £  even  if  the  actual  continuous  dynamics  deviate  from  the  model. 

A  prediction  monitor  checks  the  current  state,  because  all  previous  states  are  ensured  by  a  model 
monitor  and  subsequent  states  are  then  safe  by  (1). 

Definition  2  (^-bounded  plant  with  disturbance  d  ).  Let  apiant  be  a  model  of  the  form  x'  —  0  &  II. 

An  e-bounded  plant  with  disturbance  5,  written  oxp/an„  is  a  plant  model  of  the  form 

x0  :=  0;  (f(6, 5)  <  x'  <  g{6 ,  5)  &  H  A  x0  <  e) 

for  some  f,  g  with  fresh  variable  e  >  0  and  assuming  x'0  =  1.  We  say  that  disturbance  5  is  constant 
if  x  5;  it  is  additive  if  f(6,  5)  =  0  —  5  and  g(9,  5)  =  9  +  5. 

Theorem  3  (Prediction  monitor  correctness).  Let  a*  be  provably  safe,  i.  e.,  f=  (f>  — *  [ot*]ip  has 
been  proved  using  invariant  p  as  in  (1).  Let  Vp  =  BV(a)  U  FV([a]p).  Let  v  \=  <j>\  const  A  <p,  as 
checked  by  Xm  from  Theorem  1.  Further  assume  v  such  that  (y,  v)  G  p(actri),  as  checked  by  Xc 
from  Theorem  2.  If  (u,  V)  |=  xP  with  xP  =  (fi\ const  A  ip)  (actri)( A  [as piani\T>),  then  we  have 
for  all  {v,u)  G  p{aSpiant )  that  u  (=  ip. 

Remark  2.  By  adding  a  controller  execution  ( actri )  prior  to  the  disturbed  plant  model,  we  synthe¬ 
size  prediction  monitors  that  take  the  actual  controller  decisions  into  account.  For  safety  purposes, 
we  could  just  as  well  use  a  monitor  definition  without  controller  xP  =  (4>\ const  A  <p)  — >  [aspiantlT- 
But  doing  so  results  in  a  conserx’ative  monitor,  which  has  to  keep  the  CPS  safe  without  knowledge 
of  the  actual  controller  decision. 
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Table  2:  Monitor  complexity  case  studies 


Case  Study 

Model 

Monitor 

Time/Mem. 

dim.  proof  size 

dim.  steps  (open  seq.) 

proof  steps 

size 

([s]/[MB]) 

(branches) 

w  /  Opt.  1 

auto 

(branches) 

Water  tank 

5  38  (4) 

3  16(2) 

20  (2) 

64  (5) 

32 

2.2/45.3 

f  Cruise  control  [18] 

11  969(124) 

7  127(13) 

597  (21) 

19514(1058) 

mi 

42.8/54.9 

Speed  limit  [23] 

9  410(30) 

6  487  (32) 

5016(126) 

64311  (2294) 

19850 

239.1  /49.7 

Water  tank 

5  38  (4) 

1  12(2) 

14(2) 

40  (3) 

20 

1.3/24.6 

Cruise  control  [18] 

11  969(124) 

7  83  (13) 

518  (106) 

5840  (676) 

84 

16.4  /-’ 

£  Robot  [24] 

14  3350 (225) 

11  94(10) 

1210(196) 

26166 (2854) 

121 

39.2 /-1 

ETCS  safety  [35] 

16  193(10) 

13  162(13) 

359  (37) 

16770  (869) 

153 

14.8 /-1 

p  Water  tank 

8  80  (6) 

1  135(4) 

N/A 

307  (12) 

43 

16.7/47.7“ 

http : / / www . cs . emu . edu/ ~smitsch/ resource/modelplex_study . zip 


1  No  memory  consumption  recorded  11  Not  automated,  replaying  the  proof  containing  manual  steps 


3.6  Decidability  and  Computability 

One  useful  characteristic  of  ModelPlex  beyond  soundness  is  that  monitor  synthesis  is  computable, 
which  yields  a  synthesis  algorithm,  and  that  the  correctness  of  those  synthesized  monitors  w.r.t. 
their  specification  is  decidable,  cf .  Theorem  4. 

Theorem  4  (Monitor  correctness  is  decidable  and  monitor  synthesis  computable).  We  assume 
canonical  models  of  the  form  a  =  actrh  ai>iam  without  nested  loops,  with  solvable  differential 
equations  in  opianl  and  disturbed  plants  aspiant  with  constant  additive  disturbance  5  (see  Def  2). 
Then,  monitor  correctness  is  decidable,  i.e.,  the  formulas  Xm  (a)  Yy,  \c  (pari)Yy,  and 
XP  —>  (a)(Ty  A  \ot5piant\f)  are  decidable.  Also,  monitor  synthesis  is  computable,  i.  e.,  the  functions 
synth m  :  (a) T+  i-a  Xm,  synth  c  :  (actH) T+  (->•  yc,  and  synthp  :  (a)( T+  A  [a5plan]4>)  f-A  xP  are 
computable. 

4  Evaluation 

We  developed  a  software  prototype,  integrated  into  our  modeling  tool  Sphinx  [25],  to  automate 
many  of  the  described  steps.  The  prototype  generates  ym,  yc,  and  yp  conjectures  from  hybrid 
programs,  collects  open  sequents,  and  interacts  with  KeYmaera  [34], 

To  evaluate  our  method,  we  created  monitors  for  prior  case  studies  of  non-deterministic  hy¬ 
brid  models  of  autonomous  cars,  train  control  systems,  and  robots  (adaptive  cruise  control  [18], 
intelligent  speed  adaptation  [23],  the  European  train  control  system  [35],  and  ground  robot  colli¬ 
sion  avoidance  [24]).  Table  2  summarizes  the  evaluation.  For  the  model,  we  list  the  dimension  in 
terms  of  the  number  of  function  symbols  and  state  variables,  and  the  size  of  the  safety  proof  (i.  e., 
number  of  proof  steps  and  branches).  For  the  monitor,  we  list  the  dimension  of  the  monitor  con¬ 
jecture  in  terms  of  the  number  of  variables,  compare  the  number  of  steps  and  open  sequents  when 
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deriving  the  monitor  using  manual  proof  steps  to  apply  Opt.  1  and  fully  automated  w/o  Opt.  1,  and 
the  number  of  steps  in  the  monitor  correctness  proof.  Finally,  we  list  the  monitor  size  in  terms 
of  arithmetic,  comparison,  and  logical  operators  in  the  monitor  formula.  Although  the  number  of 
steps  and  open  sequents  differ  significantly  between  manual  interaction  for  Opt.  1  and  fully  au¬ 
tomated  synthesis,  the  synthesized  monitors  are  logically  equivalent.  But  applying  Opt.  1  usually 
results  in  structurally  simpler  monitors,  because  the  conjunction  over  a  smaller  number  of  open 
sequents  (cf.  Table  2)  can  still  be  simplified  automatically.  The  model  monitors  for  cruise  control 
and  speed  limit  control  are  significantly  larger  than  the  other  monitors,  because  their  size  already 
prevents  automated  simplification  by  Mathematica.  As  future  work,  KeYmaera  will  be  adapted  to 
allow  user-defined  tactics  in  order  to  apply  Opt.  1  automatically.  The  last  column  lists  duration 
and  memory  consumption  for  automated  monitor  synthesis  in  KeYmaera  without  Opt.  1.  Find¬ 
ing  ModelPlex  and  PredictPlex  monitors  is  quite  challenging,  in  comparison  to  finding  Simplex 
monitors,  because  of  the  additional  plant  model  with  mostly  non-trivial  differential  equations.  We 
further  simulated  monitors  in  Mathematica.  The  simulation  results  are  discussed  in  App.  E. 


5  Related  Work 

Runtime  verification  and  monitoring  for  finite  state  discrete  systems  has  received  significant  at¬ 
tention  (e.  g.,  [9,  14,  21]).  Other  approaches  monitor  continuous-time  signals  (e.  g.,  [10,  26]).  We 
focus  on  hybrid  systems  models  of  CPS  to  combine  both. 

Several  tools  for  formal  verification  of  hybrid  systems  are  actively  developed  (e.  g.,  SpaceEx  [12], 
dReal  [13],  extended  NuSMV/MathSat  [6]).  For  monitor  synthesis,  however,  ModelPlex  crucially 
needs  the  rewriting  capabilities  and  flexibility  of  (nested)  [a]  and  (a)  modalities  in  d£  [29]  and 
KeYmaera  [34];  it  is  thus  an  interesting  question  for  future  work  if  other  tools  could  be  adapted  to 
ModelPlex. 

Runtime  verification  is  the  problem  of  checking  whether  or  not  a  trace  produced  by  a  program 
satisfies  a  particular  formula  (cf.  [16]).  In  [40],  a  method  for  runtime  verification  of  LTL  formulas 
on  abstractions  of  concrete  traces  of  a  flight  data  recorder  is  presented.  The  RV  system  for  Java  pro¬ 
grams  [20]  predicts  execution  traces  from  actual  traces  to  find  concurrency  errors  offline  (e.  g.,  race 
conditions)  even  if  the  actual  trace  did  not  exhibit  the  error.  We,  instead,  use  prediction  on  the  basis 
of  disturbed  plant  models  for  hybrid  systems  at  runtime  to  ensure  safety  for  future  behavior  of  the 
system  and  switch  to  a  fail-safe  fallback  controller  if  necessary.  Adaptive  runtime  verification  [4] 
uses  state  estimation  to  reduce  monitoring  overhead  by  sampling  while  still  maintaining  accuracy 
with  Hidden  Markov  Models,  or  more  recently,  particle  filtering  [15]  to  fill  the  sampling  gaps.  The 
authors  present  interesting  ideas  for  managing  the  overhead  of  runtime  monitoring,  which  could 
be  beneficial  to  transfer  into  the  hybrid  systems  world.  The  approach,  however,  focuses  purely  on 
the  discrete  part  of  CPS. 

The  Simplex  architecture  [36]  (and  related  approaches,  e.g.,  [1,  3,  17])  is  a  control  system 
principle  to  switch  between  a  highly  reliable  and  an  experimental  controller  at  runtime.  Highly 
reliable  control  modules  are  assumed  to  be  verified  with  some  other  approach.  Simplex  focuses 
on  switching  when  timing  faults  or  violation  of  controller  specification  occur.  Our  method  com¬ 
plements  Simplex  in  that  (i)  it  checks  whether  or  not  the  current  system  execution  fits  the  entire 
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model,  not  just  the  controller;  (ii)  it  systematically  derives  provably  correct  monitors  for  hybrid 
systems;  (iii)  it  uses  prediction  to  guarantee  safety  for  future  behavior  of  the  system. 

Further  approaches  with  interesting  insights  on  combined  verification  and  monitor/controller 
synthesis  for  discrete  systems  include,  for  instance,  [2,  11]. 

Although  the  related  approaches  based  on  offline  verification  derive  monitors  and  switching 
conditions  from  models,  none  of  them  validates  whether  or  not  the  model  is  adequate  for  the 
current  execution.  Thus,  they  are  vulnerable  to  deviation  between  the  real  world  and  the  model.  In 
summary,  this  paper  addresses  safety  at  runtime  as  follows: 

•  Unlike  [36],  who  focus  on  timing  faults  and  specification  violations,  we  propose  a  systematic 
principle  to  derive  monitors  that  react  to  any  deviation  from  the  model. 

•  Unlike  [4,  15,  17,  20],  who  focus  on  the  discrete  aspects  of  CPS,  we  use  hybrid  system 
models  with  differential  equations  to  address  controller  and  plant. 

•  Unlike  [17,  36],  who  assume  that  fail-safe  controllers  have  been  verified  with  some  other 
approach  and  do  not  synthesize  code,  we  can  use  the  same  technical  approach  (d£)  for 
verifying  controllers  and  synthesizing  provably  correct  monitors. 

•  ModelPlex  combines  the  leight-weight  monitors  and  runtime  compliance  of  online  runtime 
verification  with  the  design  time  analysis  of  offline  verification. 

•  ModelPlex  synthesizes  provably  correct  monitors,  certified  by  a  theorem  prover 

•  To  the  best  of  our  knowledge,  our  approach  is  the  first  to  guarantee  that  verification  results 
about  a  hybrid  systems  model  transfer  to  a  particular  execution  of  the  system  by  verified 
runtime  validation.  We  detect  deviation  from  the  verified  model  when  it  first  occurs  and, 
given  bounds,  can  guarantee  safety  with  fail-safe  fallback.  Other  approaches  (e.g.,  [3,  17, 
36])  assume  the  system  perfectly  complies  with  the  model. 


6  Conclusion 

ModelPlex  is  a  principle  to  build  and  verify  high-assurance  controllers  for  safety-critical  computer¬ 
ized  systems  that  interact  physically  with  their  environment.  It  guarantees  that  verification  results 
about  CPS  models  transfer  to  the  real  system  by  safeguarding  against  deviations  from  the  verified 
model.  Monitors  created  by  ModelPlex  are  provably  correct  and  check  at  runtime  whether  or  not 
the  actual  behavior  of  a  CPS  complies  with  the  verified  model  and  its  assumptions.  Upon  noncom¬ 
pliance,  ModelPlex  initiates  fail-safe  fallback  strategies.  In  order  to  initiate  those  strategies  early 
enough,  ModelPlex  uses  prediction  on  the  basis  of  disturbed  plant  models  to  check  safety  for  the 
next  control  cycle.  This  way,  ModelPlex  ensures  that  verification  results  about  a  model  of  a  CPS 
transfer  to  the  actual  system  behavior  at  runtime. 

Future  research  directions  include  extending  ModelPlex  with  advanced  d C  proof  rules  for  dif¬ 
ferential  equations  [31],  so  that  differential  equations  without  polynomial  solutions,  as  we  currently 
handle  for  prediction  monitor  synthesis,  can  be  handled  for  model  monitor  synthesis  as  well.  An 
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interesting  question  for  certification  purposes  is  end-to-end  verification  from  the  model  to  the  final 
machine  code. 
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A  Proofs 


A.l  Formal  Semantics  of  dC 

ModelPlex  bases  on  a  reachability  relation  semantics  instead  of  trace  semantics  [29],  since  it  is 
easier  to  handle  and  suffices  for  checking  at  sample  points. 

The  semantics  of  d£,  as  defined  in  [27],  is  a  Kripke  semantics  in  which  states  of  the  Kripke 
model  are  states  of  the  hybrid  system.  Let  M.  denote  the  set  of  real  numbers.  A  state  is  a 
map  v  :  V  — \  R;  the  set  of  all  states  is  denoted  by  Sta.  We  write  v  \=  <f>  if  formula  <p  is  true 
at  state  v  (Def.  4).  Likewise,  \9\v  denotes  the  real  value  of  term  0  at  state  v.  The  semantics  of 
HP  a  is  captured  by  the  state  transitions  that  are  possible  by  running  a.  For  continuous  evolutions, 
the  transition  relation  holds  for  pairs  of  states  that  can  be  interconnected  by  a  continuous  flow 
respecting  the  differential  equation  and  invariant  region.  That  is,  there  is  a  continuous  transition 
along  x'  =  9  k,  H  from  state  v  to  state  u,  if  there  is  a  solution  of  the  differential  equation  x'  =  9 
that  starts  in  state  v  and  ends  in  c o  and  that  always  remains  within  the  region  H  during  its  evolution. 


Definition  3  (Transition  semantics  of  hybrid  programs).  The  transition  relation  p  specifies  which 
state  u  is  reachable  from  a  state  v  by  operations  of  a.  It  is  defined  as  follows. 

1.  (y,u)  G  p(x:=9)  iff\z\v  =  [z\ufa.  z  ^  x  and  [a;]w  =  [0]„. 

2.  (v,u)  G  p(x:=*)  iff\z\v  =  [z\ufa.  z  ±  x. 

3.  (a,u)  G  p(J(p)  iff  a  =  c o  and  v  |=  fi. 

4.  (a,  uj)  G  p(x\  —  9i, . . . ,  x'n  —  9n  &  H)  iff  for  some  r  >  0,  there  is  a  (flow)  function  <^:[0,  r]  — >  Sta 

with  99(0)  =  v,cp(r)  =  l 0,  such  that  for  each  time  (  G  [0,7']:  (i)  The  differential  equation 

holds,  i.e.,  d (C)  —  [^1^(0  for  each  x%-  (H)  For  other  variables  y  {xi, . . . ,  xn}  the 
value  remains  constant,  i.e.,  tz/]^)  =  [jy]|  r-('0)-  (Hi)  The  invariant  is  always  respected,  i.e., 
ip(Q\=H. 

5.  p(a  U  /3)  —  p{ot)  U  p(/3) 

6.  p(a]  j3)  =  {(a,co)  :  (v,z)  G  p(a),(z,u)  G  p(/3)  for  a  state  z} 

7.  p(a*)  =  UneN  P(an)  where  a*+1  =  (a;  a1)  and  a0  =  Itrue. 

Definition  4  (Interpretation  of  cfC  formulas).  The  interpretation  |=  of  a  d£  formula  with  respect  to 
state  v  is  defined  as  follows. 

T  v  |=  0i  ~  02  iff{9i\v  ~  [92\vfor~  G  {=,  <,  <,  >,  >} 

2.  v  \=  f  Aip  iffv  |=  (p  and  v  |=  ip,  accordingly  for  -1,  V,  — >,  GG 

3.  v  |=  \/x  <p  iff  co  |=  (pfor  all  c 0  that  agree  with  v  except  for  the  value  of  x 
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4.  v  |=  3x  (ft  iff  u  |=  (ft  for  some  uj  that  agrees  with  v  except  for  the  value  of  x 

5.  v  \=  [a](ft  ijfu  \=  (ftVu  with  (u,u>)  £  p(a) 

6.  v  \=  ( a)(ftijfui  |=  (ft3ojwith  £  p(a) 

We  write  \ =  (ft  to  denote  that  (ft  is  valid,  i.  e.,  that  v  |=  (ft  Vza 

A.2  Soundness 

We  recall  Lemma  1. 

Lemma  1  (Loop  prior  and  posterior  state).  Let  a  be  a  hybrid  program  and  a*  be  the  program  that 
repeats  a  arbitrarily  many  times.  Assume  that  all  consecutive  pairs  of  states  ( ,  of)  €  p(a)  of 
n  £  N+  executions,  whose  valuations  are  recalled  with  T'v  =  /\xeV  x  =  xl  and  T(-  1  are  plausible 
w.r.t.  the  model  a,  i.  e.,  \=  f\]<l<ri  (T^1  — y  (a)  Y(-]  with  Yft  =  Tf  and  =  Yy.  Then,  the 
sequence  of  states  originates  from  an  a*  execution  from  Yy-  to  T  y,  i.  e.,  |=  Yy-  — y  (a*)Yy. 

Proof  Follows  from  the  transition  semantics  of  a*:  p(a*)  =  [JneN  p{aU)  where  o,+1  =  (a;  o') 
and  a0  =  Itrue.  □ 

We  recall  Theorem  1. 

Theorem  1  (Model  monitor  correctness).  Let  a*  be  provably  safe,  so  |=  (ft  — >  [a*]ift.  Let  Vrn  = 
BV (a)  U  FV  (ift).  Let  z/0,  ui,  u2,  o3  . . .  £  Mn  be  a  sequence  of  states,  with  o0  |=  (ft  and  that  agree 
on  £\14n,  i.  e.,  z/0|E\vm  =  \vmfor  all  k.  We  define  {v,  Vi+i)  |=  \m  os  Xm  evaluated  in  the  state 
resulting  from  v  by  interpreting  x+  as  vi+l(x)  for  all  x  £  Vm,  i.  e.,  ou’fl('x'1  \=  \m.  lf{yi ,  oi+i)  \=  \m 
for  all  i  <  n  then  we  have  vn\=  ift  where 

Xm  =  (0| const  ->  («)TVm)  (3) 

and  0 1 const  denotes  the  conditions  of  (ft  that  involve  only  constants  that  do  not  change  in  a,  i.  e., 
FV(<ft  |  const  )  n  BV(a)  =  0. 

Proof  By  induction  over  n.  If  n  —  0  then  (z/0,  z/0)  £  p(a*)  trivially  by  definition  of  p  and  |=  (ft  — >■ 
[a*]ift  implies  z/0  |=  ift.  For  n  >  0  assume  (u0,  vn )  £  p(a*)  and  (z/n,  un+i)  \—  (a)  /\xeVrn  x  =  x+. 
Then  there  exists  p  such  that  {o  p)  £  p(a)  and  the  two  states  agree  on  all  variables 

except  the  ones  modified  by  a,  i.e.,  vn  ^++i(;e)|s\w(q)  =  n\ s\w(a)-  Thus,  p  |=  T£m,  i.e.,  p  \= 
AxsVm  x  =  x+’  which  in  turn  yields  p(x)  =  p{x+ )  =  vn  X++I^x\x+)  =  un+i(x)  (in  other  words, 
p\vm  =  vn+i\ Vm)-  Since  also  z/n|s\vm  =  Vn+i\z\vm  we  get  p  =  vn+1  and  (un,  vn+l)  £  p{a).  Hence 
(v0,  vn+\)  £  p(a*)  because  by  induction  hypothesis  (z/0,  vn)  £  p(a*)  and  we  conclude  un+\  |=  ift 
by  assumption  |=  (ft  — *  [a*]ift  using  u0  \=  <ft. 

□ 


We  recall  Theorem  2. 
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Theorem  2  (Controller  monitor  correctness).  Let  a  of  the  canonical  form  actri ;  op/an,.  Assume 
|=  f  — »  [a*]  A  has  been  proven  with  invariant  p  as  in  (1).  Let  v  |=  0 \ C0ILS,  A  p,  as  checked  by  \m 
(Theorem  1).  Furthermore,  let  u  be  a  post-controller  state.  If  (u,u)  |=  Xc  with  Xc  =  0 \ const  — >■ 
(«c(ri)Tjc  then  we  have  that  (u,  v)  G  p(oictri )  and  P  |=  <p- 

Proo/  Consider  a  state  ;/  |=  o>|  const  A  p.  Assume  [u,  9)  |=  Xc,  i-  c.,  if'p' !  |=  Xc-  Then  there  exists 
p  such  that  (z/f,  //)  G  p(actri)  and  the  two  states  agree  on  all  variables  except  the  ones  modified 
by  actrl,  i.e.,  Ki;(+x)f\BV(aart)  =  p\s ;\w(actrl)-  Thus,  p  \=  i.e.,  p  |=  /\xeVcx  =  x+ ,  which 

in  turn  yields  p(x)  =  p(x+)  =  pc+f> (x+)  =  u(x)  (in  other  words,  p\Vc  =  b\Vc).  Since  also 
p\t,\vc  =  v\ s\yc  we  get  p  =  b  and  (u,  v)  G  p(actri).  Then  we  have  b  \ =  p  because  by  assumption 
P  — >  [actri;  Opiant] p  and  p(apiant)  is  reflexive  as  ODE  can  evolve  for  time  0.  □ 

We  recall  Theorem  3. 

Theorem  3  (Prediction  monitor  correctness).  Let  a*  be  provably  safe,  i.  e.,  |=  0  — >  [a*]ip  has 
been  proved  using  invariant  p  as  in  (1).  Let  Vp  =  BV(a )  U  FV([a\p).  Let  v  \=  (j)\comt  A  p,  as 
checked  by  Xm  from  Theorem  1.  Further  assume  b  such  that  (u,  b)  G  p(actri),  as  checked  by  Xc 
from  Theorem  2.  If  (u,  b)  \=  xP  with  xP  =  (0|  const  A  p)  -x  (acfr/)(T+  A  [aSpiant]p),  then  we  have 
for  all  {b,u)  G  p(a5piailt)  that  uj  \=  p. 

Proof  Consider  a  state  u  such  that  u  f=  0|const  A  p.  Let  b  be  some  state  such  that  (u,  b)  G 
p(ac tri).  Then  we  have  b  \ —  p  because  by  assumption  p  — >  [cectrl ;  apiant]<p  and  p(apiant)  is  reflexive 
as  ODE  can  evolve  for  time  0.  Furthermore  b  |=  0|COnst  since  u\s \w(actrl)  =  b\?,\BV(aarl)  and 
FV (0 1 const)  O  BV (actri)  =  0.  Assume  (u,  b)  \=  xp,  i.  e.,  vvx+  ]  f=  xP-  Then  there  exists  p  such  that 
p  f=  T  y  A  [a^piant]  p  with  (ipJ} ,  p)  G  p(actr  i)  and  the  two  states  agree  on  all  variables  except  the 

ones  modified  by  actri,  i.e.,  ^|x)|s\BV(actri)  =  d\ s\BV(actrl)-  Thus,  p{x)  =  p(x+)  =  vx+\x+)  = 
b(x).  (in  other  words,  p\vv  =  d\vp)-  However,  from  xP  we  know  that  p  |=  [a,5piant]<p.  Thus,  by  the 
coincidence  lemma  [29,  Lemma  2.6]  b  |=  [a,5piant]<p  since  FV ([aPp]Linl]p)  C  Vp  and  hence  we  have 
uj  |=  p  for  all  ( v ,  uS)  G  p(a5pia„t).  □ 

Observe  that  this  is  also  true  for  all  intermediate  times  (  G  [0,  oj(t)\  by  the  transition  semantics 
of  differential  equations,  where  u(t)  <  e  because  Opp|anL  is  bounded  by  e. 

A.3  Decidability  and  Computability 

From  Lemma  1  it  follows  that  online  monitoring  [16]  (i.e.,  monitoring  the  last  two  consecutive 
states)  is  permissible.  So,  ModelPlex  turns  questions  [a*]0  and  (a*)0  into  [a]0  and  (a)0,  respec¬ 
tively.  For  decidability,  we  first  consider  canonical  hybrid  programs  a  of  the  form  a  =  actri;  apiant 
where  actri  and  aplant  are  free  of  further  nested  loops. 

We  split  Theorem  4  (decidability  and  computability)  into  Theorem  5  (decidability)  and  Theo¬ 
rem  6  (computability)  and  prove  them  separately.  To  handle  differential  inequalities  in  d£  formulas 
of  the  form  |appiam|0,  the  subsequent  proofs  additionally  assume  the  rules  for  handling  differential- 
algebraic  equations  in  the  d£  calculus  [29]. 
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Theorem  5  (Monitor  correctness  is  decidable).  Monitor  correctness  is  decidable  for  canonical 
models  of  the  form  a  =  actri ;  ctpiant  without  nested  loops,  with  solvable  differential  equations  in 
otpiant  and  disturbed  plants  a$piant  with  constant  additive  disturbance  5,  i.  e.,  \m  — »  {a)Ty,  \c 
(actri)T^,  and  xP  (a)(T£  A  [aspiant]^)  are  decidable. 

Proof  From  relative  decidability  of  d£  [31,  Theorem  11]  we  know  that  sentences  of  d£  (i.  e., 
d C  formulas  without  free  variables)  are  decidable  relative  to  an  oracle  for  discrete  loop  invari¬ 
ants/variants  and  continuous  differential  invariants/variants.  Since  neither  «ctrl  nor  apiant  contain 
nested  loops,  we  manage  without  an  oracle  for  loop  invariants/variants.  Further,  since  the  dif¬ 
ferential  equation  systems  in  ctplant  are  solvable,  we  have  an  effective  oracle  for  differential  invari¬ 
ants/variants.  Let  C/v(</>)  denote  the  universal  closure  of  d£  formula  A  (i.  e.,  CF(o)  =  V2eFv(^)^-0)- 
Note  that  when  |=  F  then  also  |=  Cl-fF)  by  a  standard  argument. 

Model  monitor  Xm  (ct)Ty:  Follows  from  relative  decidability  of  dC  [31,  Theorem  11],  be¬ 
cause  CZv(xm  — *  (a)Ty)  contains  no  free  variables. 

Controller  monitor  Xc  (oicLr[)  Tv. ;  Follows  from  relative  decidability  of  d£  [31,  Theorem  11], 
because  C7v(Xc  — >  (actri)Ty)  contains  no  free  variables. 

Prediction  monitor  xp  -A  (« ctri)(Ty  A  [ct«5piant]</>):  Decidability  for  actri  follows  from  case  Xc  — >■ 
(actri)Ty  (controller  monitor)  above.  It  remains  to  show  decidability  of  xP  —>  (actri)  [aslant]  0, 
which  by  decidability  of  the  controller  monitor  is  (xp  A  Ty)  — >■  [a^piaiu|  A-  Since  the  distur¬ 
bance  5  in  a,5piant  is  constant  additive  and  the  differential  equations  in  op|ant  are  solvable, 
we  have  the  disturbance  functions  f(9,  3)  and  g ( 9 , 3)  applied  to  the  solution  as  an  ora¬ 
cle5  for  differential  invariants  (i.  e.,  the  differential  invariant  is  a  pipe  around  the  solution 
without  disturbance).  Specifically,  to  show  (xP  A  Ty)  -a  [a<5piant]0  by  Def.  2  we  have  to 
show  (xp  A  Ty)  — >  [x0  :=  0;  {6  —  5  <  x'  <  6  +  5  &  H  A  x0  <  e}]4>.  We  proceed  with  only 
(xp  A  Ty)  — >■  [x0  :=  0;  {x'  <  6  +  5  &  H  A  x0  <  e}](j)  since  the  case  9  —  5  <  x'  follows 
in  a  similar  manner.  By  definition  of  Q:()p|ant  we  know  0  <  xq,  and  hence  continue  with 
(xP  A  Ty)  — >■  \{x'  <  9  +  5  &  Ff  AO  <  x0  <  e}]4>  by  differential  cut  0  <  x0.  Using  the 
differential  cut  rule  [29],  we  further  supply  the  oracle  solx.  +  5x0,  where  sol.,,  denotes  the  so¬ 
lution  of  x'  =  9  in  G: p|ant  and  5x0  the  solution  for  the  disturbance  since  5  is  constant  additive. 
This  leads  to  two  proof  obligations: 

Prove  oracle  (xP  A  Ty)  — >■  [x'  <  9  +  5  &  0  <  x0  <  e]x  <  sol^  +  5x0,  which  by  rule  differ¬ 
ential  invariant  [29]  is  valid  if  we  can  show  0  <  x0  <  e  — >  x'  <  sol),  +  (3xo)'  where 
the  primed  variables  are  replaced  with  the  respective  right-hand  side  of  the  differential 
equation  system.  From  Def.  2  we  know  that  xr0  =  1  and  5'  =  0  and  since  sol,,  is  the 
solution  of  x’  =  9  in  aplant  we  further  know  that  sol),  =  9:  hence  we  have  to  show 
0<xo<£— +  +  which  is  trivially  true. 

Use  oracle  (xP  A  Ty)  — >  [x'  <  9  +  5  &  H  A  0  <  x0  <  £  A  x  <  solx  +  5x0}(p,  which  by  rule 
differential  weaken  [29]  is  valid  if  we  can  show 

(xP  A  Ty)  — >■  V"  ((H  AO  <  x0  <  £  A  x  <  sol^  +  5x0)  —>  <j>) 

5  By  design,  the  disturbed  plant  a5piant  also  includes  a  clock  xq,  so  the  oracle  additionally  includes  the  trivial  differ¬ 
ential  invariant  x()  >  0. 


22 


where  V"  denotes  the  universal  closure  w.r.t.  x,  i.  e.,  V.x.  But,  if  yp  is  a  correct  monitor, 
this  is  provable  by  quantifier  elimination.  Furthermore,  we  cannot  get  a  better  result 
than  differential  weaken,  because  the  evolution  domain  constraint  contains  the  oracle’s 
answer  for  the  differential  equation  system,  which  characterizes  exactly  the  reachable 
set  of  the  differential  equation  system. 

We  conclude  that  the  oracle  is  proven  correct  and  its  usage  is  decidable. 


□ 

For  computability,  we  start  with  a  theoretical  proof  on  the  basis  of  decidability,  before  we  give 
a  constructive  proof,  which  is  more  useful  in  practice. 

Theorem  6  (Monitor  synthesis  is  computable).  Synthesis  ofx,n,  Xc,  and  xP  monitors  is  computable 
for  canonical  models  of  the  form  a  =  actri]  apiant  without  nested  loops,  with  solvable  differential 
equations  in  apiant  and  plants  aspiant  with  constant  additive  disturbance  S,  i.  e.,  synth  m  :  (a)  Yp-  i— >■ 
Xm,  synth c  :  (acrr/)T^  yy,  and  synthp  :  (a)(T£  A  \aSpiant\(p)  Xp  are  computable. 

Proof  Follows  immediately  from  Theorem  5  with  recursive  enumeration  of  monitors.  □ 

We  give  a  constructive  proof  of  Theorem  6.  The  proof  is  based  on  the  observation  that,  except 
for  loop  and  differential  invariants/variants,  rule  application  in  the  d£  calculus  is  deterministic: 
from  [29,  Theorem  2.4]  we  know  that,  relative  to  an  oracle  for  first-order  invariants  and  variants, 
the  6C  calculus  gives  a  semidecision-procedure  for  dd  formulas  with  differential  equations  having 
first-order  definable  flows. 

Proof  For  the  sake  of  a  contradiction,  suppose  that  monitor  synthesis  stopped  with  some  open 
sequent  not  being  a  first-order  quantifier-free  formula.  Then,  by  [29,  Theorem  2.4]  the  open  se¬ 
quent  either  contains  a  hybrid  program  with  nondeterministic  repetition  or  a  differential  equation 
at  top  level,  or  it  is  not  quantifier-free.  But  this  contradicts  our  assumption  that  both  actri  and  apiarn 
are  free  from  loops  and  that  the  differential  equations  are  solvable  and  disturbance  is  constant,  in 
which  case  for 

Model  monitor  synthesis  Xm :  the  solution  rule  (')  would  make  progress,  because  the  differential 
equations  in  apiant  are  solvable;  and  for 

Prediction  monitor  synthesis  y,,:  the  disturbance  functions  f(9 ,  5)  and  g(6,  5)  applied  to  the  so¬ 
lution  provide  differential  invariants  (see  proof  of  Theorem  5)  so  that  the  differential  cut  rule, 
the  differential  invariant  rule,  and  the  differential  weakening  rule  [29]  would  make  progress. 

In  the  case  of  the  open  sequent  not  being  quantifier-free,  the  quantifier  elimination  rule  QE  would 
be  applicable  and  turn  the  formula  including  quantifiers  into  an  equivalent  quantifier-free  formula. 
Hence,  the  open  sequent  neither  contains  nondeterministic  repetition,  nor  a  differential  equation, 
nor  a  quantifier.  Thus  we  conclude  that  the  open  sequent  is  a  first-order  quantifier- free  formula.  □ 
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B  Water  Tank  Monitor  Specification  Conjecture  Analysis 

Proof  1  shows  a  complete  sequence  of  proof  rules  applied  to  the  water  tank  specification  conjecture 
of  Example  2  on  page  7,  with  f  =  e  >  0  and  Y+  =  x  =  x+  A  /  =  f+  A  t  =  t+ . 


B.l  Monitoring  in  the  Presence  of  Expected  Uncertainty  and  Disturbance 

Example  7.  We  start  at  the  point  where  we  have  to  handle  the  differential  inequalities.  First,  we 
eliminate  the  differential  inequalities  by  rephrasing  them  as  differential-algebraic  constraints  in 
step  (DE).  Then,  we  refine  by  instantiating  the  existential  quantifiers  with  the  worst-case  evolution 
in  step  (DR).  The  resulting  differential  equation  has  polynomial  solutions  and,  thus,  we  can  use  (') 
and  proceed  with  the  proof  as  before. 


DE 


'  b  \/XMT(3d3t(X  =  d  A  T  =  t  (t>  b  \/X  \/T{X  =  Fd  AT  =  l  -  c  Ai)  <  x  At  <  e) 
Ad  <  /d  A  1  —  c  <  t  A  0  <  x  j  ►  3d3t(X  =  d  A  d  <  Fd  AT  =  £  A  1  —  c  <T 
At  <£-aX<F  Al  —  c<T  J  A0<xAt<s) 

A0<dAt<e) 


ib... 


cj>  b  3T>0((V0<7<T(a;  +  dFT  >  0  A  7(1  -  c)  <  e)) 

AF  =  /+  A  x  +  dFT  =  x+  A  Xs  =  x+  A  T(  1  -  c)  =  t+) 

(j)  b  (/  :=  F;  xs  :=  Xs;  t  :=  0)(x'  =  fd,  t'  =  1  —  c  0  <  x  At  <  e)T+ 

1 1  . . .  UR  (f>  b  (/  :=  F\  xs  :=  Xs;  t  :=  0){3d3t(x'  =  d,  t'  =  t&zd<  fd  Al  —  c  <t  AO  <  x  At  <  e)T+ 
'  b  (/  :=  F;  xs  :=XS;  t  :=  0)(a;,  <  fd,  1  —  c<t'  k.Q  <  x  At  <  e)T+ 


As  expected,  we  get  a  more  permissive  monitor  specification.  One  conjunct  of  the  monitor 
specification  is  shown  in  C\mi)-  Such  a  monitor  specification  says  that  there  exists  a  real  flow  F, 
a  real  time  T,  and  a  real  level  Xs,  such  that  the  measured  flow  f+,  the  clock  t+,  and  the  measured 
level  x+  can  be  explained  with  the  model. 

C  Monitor  Synthesis  and  Fallback  Controller  Design 

C.l  Design-By-Contract  Monitoring 

Preconditions,  postconditions  and  invariants  are  crucial  conditions  in  CPS  design.  Monitors  for 
these  conditions  can  check  (i)  whether  or  not  it  is  safe  to  start  a  particular  controller  (i.  e.,  check 
that  the  precondition  of  a  controller  is  satisfied),  (ii)  whether  or  not  a  controller  complies  with  its 
specification  (i.  e.,  check  that  a  controller  delivers  set  values  that  satisfy  its  postcondition),  and 
( iii)  whether  or  not  the  system  is  still  within  its  safety  bounds  (i.  e.,  check  that  the  loop  invariant  of 
a*  is  satisfied). 

Precondition  and  postcondition  monitors  are  useful  to  decide  whether  or  not  it  is  safe  to  invoke 
a  controller  in  the  current  state,  and  whether  or  not  to  trust  a  controller  output.  An  invariant  monitor 
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rh0,A  rh^A 

(Ar)  - - ,  A  (Wr) 


«;)) 

(0) 

(i3) 


r  h  0  A  0,  A 

{a)(/3)(p 


r  h  a 

r  b  <j),A 


(QE) 


QE(0)  , 


((?)) 


H  A'ljj 


((:=)) 


{a;/3)(j)  (?f/)0  ( x:—6)q i 

3t>0  ((V0<t<t  (x  :=  y(t))H )  A  (x  :=  t/(£))0)  3 


((*)) 


3X(x:=X)0, 


<x'  =  6kH)(f> 

rh3X(A.t(^E^)),A 


(3r) 


(x  :=  *)0 
rh0(fl),3x0(x),A4 
T  h  3x0(x),  A 


r,$i  h  a 


r,$n  h  ^n,A 


(3a) 


3x  (x  =  0  A  0(x)) 


1  iff  (j>  =  QE(0),  0  is  a  first-order  real  arithmetic  formula,  QE(0)  is  a  quantifier-free  formula 

2  X  is  a  new  logical  variable 

3  t  and  t  are  fresh  logical  variables  and  (x  :=y(t))  is  the  discrete  assignment  belonging  to  the  solution  y  of  the 
differential  equation  with  constant  symbol  x  as  symbolic  initial  value. 

4  9  is  an  arbitrary  term,  often  a  new  (existential)  logical  variable  X. 

5  Among  all  open  branches,  free  logical  variable  X  only  occurs  in  the  branches  F,  F;  b  'J,  ,  A 


0  h  -1  <  f+  <  ^  A  x+  =  x  +  f+t+  A  f+  >  0  A  x  >  0 
Ae  >  f+  >  0  A  f+t+  +  x  >  0 

0  h  3F(-1  <F<^AF  =  /+Ax+  =  x  +  Ef+Af+>0 
Ax>0A£>f+>0A  Ft+  +  x  >  0) 

0hF  =  /+Ax+  =  x  +  Ft+  Ai+>0Ax>0 
Ae  >  t+  >  0  A  Ft+  +  x  >  0 
QE  0  h  VO <t<t+  (x  +  Ft  >  0  A  t  <  e)  A  F  =  J+ 

Ax+  =  x  +  Ft+  A  t+  =  t+ 

3r’Wr  0  h  3T>0((V0<t<T  (x  +  Ft  >  0  A  t  <  e)) 

A(F  =  f+  A  x+  =  x  +  FT  A  t+  =  T)) _ 

^  0  h  (/  :=  F;  t  :=  0)(x'  =  /,  t1  =  1  &  x  >  0  A  t  <  e)T+ 

<;>’<:=>0  h  (/  :=  F)(t  :=  0;  (x7  =  /,  £'  =  1  &  x  >  0  A  t  <  e))T+ 
i3  0h  -1  <  F  <  ^  0  b  {f  :=  F)(plant)r+ 

Ar  £0  h  (/  :=  F)  -  1  <  /  <  A  (plant) T+ 

<?>  0h  (/:=F)(?  -  1  <  /  <£^)(p/a/7f)T+ 

r’Wr  0  h  3F(/  :=  F)(?  —  1  <  /  <  ^)(p/m2f)T+ 

(*}  0h  (/:=*;?-!  </<^)(/^)T+ 

<;>  0h  (/:=*;?-!</<  mfL]plant)r+ 


Proof  1:  Analysis  of  the  water  tank  monitor  specification  conjecture  (jjlant  is  an  abbreviation  for 

x'  =  /,  t’  =  1  &  x  >  0  A  t  <  e) 


of  a  CPS  a*  captures  the  main  assumptions  that  have  to  be  true  throughout  system  execution.  When 
an  invariant  monitor  is  unsatisfied,  it  may  no  longer  be  safe  to  run  the  CPS;  a  fail-safe  controller 
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can  act  as  a  mitigation  strategy. 

Design-by-contract  monitors  are  useful  to  monitor  specific  design  decisions,  which  are  explic¬ 
itly  marked  in  the  model.  Our  approach  systematically  creates  monitors  for  a  complete  specifica¬ 
tion  of  the  behavior  of  the  model. 

C.2  Monitor  Synthesis 

Once  we  found  a  model  monitor,  controller  monitor,  or  prediction  monitor  specification,  we  want 
to  turn  it  into  an  actual  monitor  implementation  (e.  g.,  in  C).  The  main  challenge  is  to  reliably 
transfer  the  monitor  specification,  which  is  evaluated  on  M,  into  executable  code  that  uses  floating 
point  representations.  We  use  the  interval  arithmetic  library  Apron  to  represent  each  real  arithmetic 
value  with  an  interval  of  a  pair  of  floating  point  numbers.  The  interval  reliably  contains  the  real. 

For  certification  purposes  one  still  has  to  argue  for  the  correctness  of  the  actual  machine  code 
of  the  synthesized  monitor.  This  entails  that  the  transformation  from  the  monitor  specification  as 
a  first-order  formula  into  actual  code  that  evaluates  the  formula  must  be  formally  verified.  If  the 
synthesized  code  is  still  a  high-level  language,  a  certified  compiler,  e.  g.,  CompCert6,  can  be  used 
to  produce  machine  code.  Such  a  comprehensive  proof  chain  suitable  for  certification  is  part  of 
our  ongoing  research. 

C.3  Designing  for  a  Fail-Safe  Fallback  Controller 

When  we  design  a  system  for  a  fail-safe  fallback  controller  ctrlsa fe,  it  is  important  to  know  within 
which  bounds  the  fail-safe  controller  can  still  keep  our  CPS  safe,  and  which  design  limits  we  want 
a  controller  implementation  to  obey.  The  invariant  of  a  CPS  with  the  fail-safe  fallback  controller 
describes  the  safety  bounds.  When  we  start  the  fail-safe  fallback  controller  ctrlsaie  in  a  state  where 
its  invariant  G  is  satisfied,  it  will  guarantee  to  keep  the  CPS  in  a  state  that  satisfies  the  safety 
property  ip. 

So,  to  safely  operate  an  experimental  controller  ctrlexp,  we  want  a  monitor  that  informs  us  when 
the  experimental  controller  can  no  longer  guarantee  the  invariant  of  the  fail-safe  controller  or  when 
it  is  about  to  violate  the  design  limits. 

A  design  for  a  CPS  with  a  fail-safe  fallback  controller,  therefore,  involves  proving  two  proper¬ 
ties.  First,  we  prove  that  the  fail-safe  controller  ctrlsa{e  ensures  the  safety  property  ip  as  in  formula 
(4)  below.  This  property  is  only  provable  if  we  discover  an  invariant  G  for  the  CPS  with  the 
fail-safe  controller.  Then  we  use  G  as  the  safety  condition  for  generating  a  prediction  monitor. 

(p [(cfrZsafe;  plant)*  @inv (G)]ip  (4) 

With  this  generic  structure  in  mind,  we  can  design  for  a  fallback  controller  invoked  by  a  model 
monitor  ym,  controller  monitor  yc,  or  prediction  monitor  yp.  Upon  violation  of  either  ym,  Xc,  or 
XP  by  the  actual  system  execution,  the  set  values  of  a  fail-safe  controller  are  used  instead. 

6  http://compcert.inria.fr/ 
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D  Monitor  Synthesis  Algorithm 

Algorithm  1  lists  the  ModelPlex  specification  conjecture  analysis  algorithm,  which  turns  a  spec¬ 
ification  conjecture  into  an  actual  monitor.  The  algorithm  takes  a  hybrid  system  model  a,  a  set 
of  variables  V  that  we  want  to  monitor7,  and  an  initial  condition  including  constraints  on  the 
variables  not  changed  in  a. 


Algorithm  1:  ModelPlex  monitor  synthesis 

input  :  A  hybrid  program  a,  a  set  of  variables  V  C  B V (a),  an  initial  condition  o  such  that 

h  0  ->•  [«*]A 

output:  A  monitor  ym  such  that  |=  ym  =  0| const  — >  (ct)Y+. 

begin 

S  < —  0 

T+  i —  f\  t.gV  x  =  x+  with  fresh  variables  xf  II 

G  i —  {h  0|const  -A  (ct)T+} 

l  while  G  7^0  do  //  Analyze 

foreach  g  e  G  do 

G  < —  G-{g} 
if  g  is  first-order  then 
|  if  Y=  g  then  S  < —  S  U  {g} 

else 

g  < —  apply  d C  proof  rule  to  g 

[_  Lg^-gu{^} 

Xm  < — A seSs  //  Collect  open  sequents 


Monitor  conjecture 
monitor  conjecture 


E  Simulation 

To  illustrate  the  behavior  of  the  water  tank  model  with  a  fallback  controller, we  created  two  mon¬ 
itors:  Monitor  ym  validates  the  complete  model  (as  in  the  examples  throughout  this  paper)  and  is 
executed  at  the  beginning  of  each  control  cycle  (before  the  controller  runs).  Monitor  yc  validates 
only  the  controller  of  the  model  a  (compares  prior  and  post  state  of  f  '■—*]  ?  —  1  <  /  <  )  and 

is  executed  after  the  controller  but  before  control  actions  are  issued.  Thus,  monitor  ;yc  resembles 
conventional  runtime  verification  approaches,  which  do  not  check  CPS  behavior  for  compliance 
with  the  complete  hybrid  model.  This  way,  we  detect  unexpected  deviations  from  the  model  at  the 
beginning  of  each  control  cycle,  while  we  detect  unsafe  control  actions  immediately  before  they 
are  taken.  With  only  monitor  Xm  in  place  we  would  require  an  additional  control  cycle  to  detect 
unsafe  control  actions8,  whereas  with  only  monitor  yc  in  place  we  would  miss  deviations  from  the 

7  Usually,  we  want  a  monitor  for  all  the  bound  variables  of  the  hybrid  system  model,  i.  e.,  V  =  BV (a).  8  We  could 

run  monitor  \;m  in  place  of  yc  to  achieve  the  same  effect.  But  monitor  Xm  implements  a  more  complicated  formula, 
which  is  unnecessary  when  only  the  controller  output  should  be  validated. 
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Figure  2:  Water  tank  simulation  with  monitor  illustration,  — is  maximum  level  (m),  is  current 
level  (x),  -a-  is  commanded  flow  (/),  - is  the  output  of  monitor  Xm  for  the  complete  model,  and 
■■  is  the  output  of  monitor  xc  for  the  controller 


model. 

Fig.  2  shows  a  plot  of  the  variable  traces  of  one  simulation  run.  In  the  simulation,  we  ran 
the  pump  controller  every  2  s  (e  =  2  s,  indicated  by  the  grid  for  the  abscissa  and  the  marks  on 
sensor  and  actuator  plots).  The  controller  was  set  to  pump  with  =  |  for  the  first  three 

controller  cycles,  which  is  unsafe  on  the  third  controller  cycle.  Monitor  B  immediately  detects  this 
violation  at  t  —  4,  because  on  the  third  controller  cycle  setting  f  —  |  violates  /  <  m~xi .  The 
fail-safe  action  at  t  =  4  drains  the  tank  and,  after  that,  normal  operation  continues  until  t  =  12. 
Unexpected  disturbance  x'  =  /  +  ^  occurs  throughout  t  =  [12, 14],  which  is  detected  by  monitor 
Xm •  Note,  that  such  a  deviation  would  remain  undetected  with  conventional  approaches  (monitor 
Xc  is  completely  unaware  of  the  deviation).  In  this  simulation  run,  the  disturbance  is  small  enough 
to  let  the  fail-safe  action  at  t  =  14  keep  the  water  tank  in  a  safe  state. 
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